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Abstract 

Items on achievement tests are designed to be equivalent in educational testing situations. That is, 
the information provided to the student is designed to be the same regardless of the ethnicity of the 
examinee. As a result, students of equal ability would be expected to select the same answer 
regardless of their ethnicity. However, aboriginals and non-aboriginals may use different cognitive 
processing skills, and consequently, may find items differentially easy or difficult depending on which 
cognitive processing style is elicited by the item. The present study used differential bundle 
functioning (DBF) analyses with cognitive processing theory as the organizing principle to study 
group differences. Aboriginal and non-aboriginal students were compared on Grade 6 Mathematics, 
Science, and Social Studies achievement tests. Items on these tests were identified by raters 
trained in cognitive processing theory as eliciting either simultaneous or successive cognitive 
processing skills. Items were then grouped and tested for DBF using the statistical program 
SIBTEST. It was hypothesized that differences favoring aboriginals would occur for those bundles 
eliciting simultaneous processes while differences favoring non-aboriginals would occur for those 
bundles eliciting successive processes. When the bundles were tested, performance differences did 
not favor aboriginals for simultaneous items or non-aboriginals for successive items. Ability 
distribution differences between the aboriginal and non-aboriginal samples coupled with small 
sample sizes for the aboriginal samples were key limitations of the study. Recommendations for 
future research are presented. 
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Differential Bundle Functioning on Three Achievement Tests: A Comparison of 
Aboriginal and Non-Aboriginal Examinees 

The use of large-scale achievement testing is pervasive in North America. Achievement tests are 
used by school districts to make decisions about individual students, to track students, classes, and 
schools as well as to determine the promotion of students from one grade to the next. The Canadian 
province of Alberta, for example, tests students at Grades 3, 6, and 9 in the content areas of 
Language Arts, Mathematics, Science, and Social Studies. These tests are administered to 
approximately 40 000 students annually and the test results are used by many teachers to evaluate 
students. A portion of the students writing these tests are of aboriginal descent, as Alberta has one 
of the largest native student populations in Canada. Traditionally, aboriginal students have 
demonstrated lower levels of academic achievement when compared to national norms. Anderson 
and Postl (2000) found that fewer aboriginal than non-aboriginals students meet a passing standard 
across content areas and grade levels on British Columbia provincial achievement tests. They also 
noted that this pattern first appears during the early years of schooling and continues as the students 
progress through school. 

Validity is an important topic in achievement testing because test scores have consequences for 
students. One of the primary validity issues in achievement testing is bias. Statistical methods are 
used in conjunction with judgmental methods to detect bias. The prominent statistical methods used 
by psychometricians to detect bias are generally described as differential item functioning (DIF) 
analyses (see review by Clauser & Mazor, 1998). DIF first involves dividing examinees into two 
separate groups — the reference and focal group. Typically, examinees from the focal group are 
viewed as being disadvantaged on the test. The reference group examinees serve as the standard 
of comparison. Examinees from each group are then matched according to their ability derived from 
the test and statistical procedures are used to estimate group differences. An item is deemed to 
exhibit DIF when examinees from two groups have a different probability of answering an item 
correctly after controlling for overall ability. 

DIF can be attributed to impact or bias. Item impact is present when there is a difference in the 
probability of answering an item correctly between two groups, which reflects an actual difference in 
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knowledge between the groups on the construct of interest. In contrast, item bias reflects systematic 
error in how an item measures the construct of interest for one of the two groups. When an item 
unfairly favors one group of examinees over another the item is deemed to be biased. Typically, 
items identified as exhibiting DIF are reviewed by groups of specialists (e.g., test developers and 
content specialists) who attempt to provide a substantive interpretation of the cause of the DIF. 
Roussos and Stout (1996) note that this form of DIF analysis lacks power due the exploratory nature 
of a single-item analysis. Moreover, these reviews have largely been unsuccessful at discerning the 
causes of DIF. 

Two recent advances have allowed researchers to overcome the limitations of single-item 
exploratory DIF analysis. Confirmatory differential bundle functioning (DBF), in contrast to 
exploratory DIF, involves generating substantive hypotheses prior to the statistical analysis and 
testing items as a bundle or group of items. Generally, substantive DBF analysis requires a way of 
linking item characteristics to the statistical outcomes (Roussos & Stout, 1996a). Bundles are 
formed based on organizing principles such as the use of test specifications, content, empirical, or 
psychological analysis. Organizing items based on psychological analysis involves the use of a 
hypothesized structure formulated from a psychological perspective or theory (Gierl, Bisanz, Bisanz, 
Boughton & Khaliq, 2001). In recent years there has been a shift in psychology away from 
behavioral theory and towards cognitive theory. Cognitive psychology is the study of internal mental 
processes in contrast to overtly displayed behavior (Reber, 1995). The current study uses cognitive 
style and, more specifically, Planning Attention Simultaneous and Successive (PASS) theory, in an 
effort to provide insight into the achievement differences between aboriginal and non-aboriginal 
students. 

Researchers have found that achievement scores for aboriginal examinees are consistently lower 
than those of non-aboriginals. To date, however, researchers have failed to address the role 
cognitive processes may play in this outcome. The goal of the present study was to examine the 
performance of aboriginal and non-aboriginal students on Grade 6 Mathematics, Science, and Social 
Studies achievement tests. The primary research question examined in the current study is: Are 
there differences favoring aboriginals for bundles of items eliciting simultaneous processes? 
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Although there is a consistent pattern of poor performance for aboriginals on achievement measures 
compared to their non-aboriginal counterparts, it is unclear what factors are influencing this 
difference. The information obtained from the current study could be of practical benefit to future test 
developers and their attempts to minimize the effects of DIF. By determining the cognitive 
processing skills used by aboriginal children this information may help educators teach and evaluate 
these students. Janzen (2000) argues: 

If research confirms that native children tend to prefer simultaneous modes of information 
processing to successive processing then educators are faced with three basic options: 

First, educators may be advised to use primarily visual modes of instruction and try to 
provide an overall picture of a subject. Second, one could start encouraging early activity 
that promotes the development of successive processing skills. Third, one could utilize a 
combined approach that emphasizes both modes of processing equally, (p. 4) 

Of course, this line of reasoning is only valid if aboriginal students do, in fact, use simultaneous 
modes of information processing consistently. Few studies, with the exception of Janzen (2000) and 
Krywaniuk (1974), have examined the use of the PASS model with aboriginals. We begin with a 
review of PASS theory followed by a summary of the research on cognitive styles for aboriginal 
students. 

Literature Review 

Simultaneous and Successive Theory 

Luria (1966) divided the brain into three functional units. These functional units were established 
on both a conceptual basis as well as through Luna's clinical examinations of individuals with cortical 
lesions. The first functional unit involves arousal and attention and is located in the primitive brain 
stem, the reticular formation, limbic cortex, and hippocampus (Das, Kirby & Jarman, 1975). The 
second unit, of primary interest to the current study, involves the input, recoding, storing, and 
integration of information. This unit is located in the occipital, parietal, and frontal-temporal regions 
of the brain. The final unit is involved in the planning and programming behavior and falls under the 
control of the frontal lobes. Das, Naglieri and Kirby (1994) have since expanded Luria's three 
functional units into the Planning, Attention, Simultaneous and Successive (PASS) theory. 
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The current study focuses on differences in the way aboriginals and non-aboriginals may process 
information. Consequently, the focus will be on the simultaneous and successive processing 
components of the theory. Luria believed simultaneous processing was housed within the parietal- 
occipital lobes while successive processing fell under the direction of the fronto-temporal lobes. 
Simultaneous processing is evident when there is a recognition of patterns among units of 
information. The resulting code takes up less space because various units of information are 
perceived as a whole rather than individual pieces of information. The order the information was 
received becomes unimportant after coding has occurred. As a result, an individual may remember 
a category of information but may not recall the individual pieces of information within that category. 
Simultaneous tasks tend to be presented visually and involve activities such as figure copying, verbal 
analogies, and spatial relations. In contrast, successive processing involves processing information 
in a sequential order where the retention of the order is important. The information takes up as much 
space as the number of items of information in the code, rather than being integrated into a whole as 
is the case in simultaneous processing. Successive tasks tend to be presented auditorily and 
include activities such as serial and free recall. 

An everyday example that highlights the key differences between the two forms of processing, is 
learning a new phone number. Often, when individuals are first presented with a novel phone 
number, they will read and memorize each number in a successive fashion from left to right. In 
contrast, other individuals may recognize that the new number spells a word thereby using 
simultaneous processing skills. The operation of the PASS model, and more specifically the 
Simultaneous and Successive processing systems, is depicted in Figure 1. As illustrated, 
information can enter (input) or leave (output) the systems through any one of the senses. 
Additionally, the magnitude of the arrows illustrates the strength of the relationship between the 
functional units. As illustrated, there is a greater relationship between the Planning and Attention 
systems than there is between the Simultaneous and Successive unit and the remaining 
components of the model. 

Lesser (1976) stated that individuals who share a common cultural background also share 
common patterns of cognitive styles. Cognitive style refers to a habitual mode of processing 
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Information that is adopted by a group (Das, Manos & Kanungo, 1975). Some researchers have 
suggested that aboriginals may have a unique preference for specific modes of information 
processing (Larose, 1991; More, 1987; Sawyer, 1991; Swisher & Deyhle, 1989; Wauters, Bruce, 
Black & Hocker, 1989). Researchers have found that, in contrast to non-aboriginal children, 
aboriginal children are more likely to use simultaneous cognitive processes rather than successive 
processing skills in some situations (Janzen, 2000; Kaufman & Kaufman, 1983; Krywaniuk, 1974; 
Krywaniuk & Das, 1976; More, 1984). For example, Krywaniuk and Das (1976) found that Cree 
aboriginals tended to integrate new information with previous knowledge via symbolic relationships. 
Janzen (2000) found that Cree aboriginals performed best on simultaneous tasks of the Cognitive 
Assessment System (CAS) such as recalling abstract figures. Conversely, they demonstrated 
significant delays on successive processing tasks relative to the normative group on the test. These 
research results, although far from complete, suggest that aboriginal and non-aboriginal examinees 
may process information differently. If these findings are accurate, then some performance 
differences on achievement tests may be attributed to cognitive style where aboriginal examinees 
excel at items eliciting simultaneous processes and non-aboriginal examinees excel at items eliciting 
successive processes. We set out to evaluate this hypothesis. 

Methods 

Student Sample and Achievement Tests 

Data from six different student samples in three content areas at one grade level were analyzed 
in the current study. Analyses were conducted using the results from Mathematics, Science, and 
Social Studies achievement tests administered at Grade 6 during the 1999 school year in the 
Canadian province of Alberta. Each test consisted of 50 multiple-choice items, with each item 
containing four options. Mathematics, Science, and Social Studies tests were chosen because they 
are contrasting content areas — these areas have different content requirements and therefore may 
elicit different cognitive processes. The sample consisted of 2956 Mathematics students (956 
aboriginal and 2000 non-aboriginal), 2480 Science students (480 aboriginal and 2000 non- 
aboriginal), and 2971 Social Studies students (971 aboriginal and 2000 non-aboriginal). The 
aboriginal sample included students who wrote the achievement tests in English and for whom the 
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educating school authority received payment of a tuition fee from the Government of Canada. The 
Department of Indian Affairs and Northern Development pays this tuition for elementary educational 
services for Registered Indian children. As such, Non-Status Indians, Inuit, and Metis children were 
excluded from the aboriginal sample. The non-aboriginal sample was randomly selected from all 
schools, excluding those schools that were band operated or students for whom the educating 
school authority received payment of the tuition fee. Students that were in special programs such as 
English as a Second Language, special education, home education, French immersion or who used 
special provisions to write the achievement test were excluded from both the aboriginal and non- 
aboriginal samples. 

Item Selection 

Based on the descriptions of simultaneous and successive processing by Luria (1966) and Das, 
Naglieri and Kirby (1994), four doctoral students trained in processing theory by Professor. J. P. Das 
at the University of Alberta classified items as eliciting either a simultaneous or successive 
processing style. Items were coded based on their most salient characteristics, as no task is purely 
simultaneous or successive (Krywaniuk & Das, 1976). Additionally, raters were provided with a list 
of characteristics and examples of simultaneous and successive items from the research literature. 
Initially, all items were independently rated. Then, the raters meet to discuss their classifications and 
reach consensus on the type of processing style elicited by each item. Raters were also asked to 
provide a confidence rating for their classification on a 3-point scale, ranging from 1 (Not Confident) 
to 3 (Very Confident). Those items with a confidence rating of 3 that were classified as eliciting a 
simultaneous processing style were hypothesized to favor aboriginal examinees. Conversely, those 
items with a confidence rating of 3 that were classified as eliciting a successive processing style 
were hypothesized to favor non-aboriginal examinees. While raters were able to reach consensus in 
their classifications of simultaneous and successive processes for all 50 items of the Science and 
Social Studies tests, they failed to do so for four of the 50 items on the Mathematics test. These four 
items were dropped from the study. 

On the Mathematics test, 10 items were expected to elicit simultaneous processes (therefore 
favor aboriginals). In contrast, three items were expected to elicit successive processes (therefore 
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favor non-aboriginals). On the Science test, 14 simultaneous processing items were identified while 
four successive processing items were coded. On the Social Studies test, six simultaneous 
processing items were coded while two successive processing items were identified. Raters 
appeared to have greater ease in classifying items on the Science test followed by the Mathematics 
test. In contrast, raters found it difficult to identify items as eliciting simultaneous or successive 
processing on the Social Studies test as demonstrated by the low number of items on this test with 
confidence ratings of 3. This result is not surprising as many of the features of simultaneous and 
successive processes (e.g., geometry, word problems, and number fact knowledge) are more overtly 
apparent in Mathematics and Science compared to Social Studies. These groups of items formed 
the “suspect” or “studied” subtest for the DBF analyses. The remaining items served as the “valid” or 
“matching” subtest. 

Statistical Analysis 

Once the raters completed their substantive reviews, item and bundle analyses were performed. 
The Simultaneous Item Bias Test (SIBTEST) was used to test the data from the aboriginal and non- 
aboriginal examinees in each of the content areas at both the item and bundle level. First, items 
were tested with a single-item confirmatory DIF analysis to determine if the items predicted to favor 
aboriginals and non-aboriginals did, indeed, favor these groups. SIBTEST produces an overall 

statistical test as well as a measure of the effect size for each item ( fi UNI ). Roussos and Stout 
(1996b) presented the following guidelines to interpret single-item SIBTEST results: 

• Negligible or A-level DIF: Null hypothesis is rejected and the absolute value of fi UNI < 0.059, 

• Moderate or B-level DIF: Null hypothesis is rejected and 0.059 <1 ft UNI I <0.088, and 

• Large or C-level DIF: Null hypothesis is rejected and I fi UNI \ >0.088. 

These guidelines are used to classify DIF items in the present study. A positive fi UNI indicates DIF 

in favor of the reference group (non-aboriginals) while a negative fi UNI value indicates DIF in favor 
of the focal group (aboriginals). Second, items were grouped and used in a confirmatory DBF 
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analyses. All hypotheses were tested with a directional test assuming that simultaneous items with 
confidence ratings of 3 would favor aboriginals and successive items with confidence ratings of 3 
would favor non-aboriginals. Because few items received a confidence rating of 1, the matching 
subtests for these analyses were formed from those simultaneous or successive items with 

A 

confidence ratings of 1 or 2. Similar to the single-item analysis, a positive ( 3 ^ value indicates that 

the bundle favors non-aboriginals while a negative value indicates that the bundle favors aboriginals. 
In contrast to single-item DIF outcomes, there are no guidelines to classify bundles as A-, B-, or C- 
level. Instead, statistical outcomes were used to interpret the results. All statistical tests were 
interpreted using an alpha level of 0.05. 

Results 

Psychometric Characteristics of the Achievement Tests 

A summary of the observed psychometric characteristics on the Mathematics, Science, and 
Social Studies achievement tests for aboriginal and non-aboriginal examinees is presented in Table 
1. Some general trends are highlighted here. The psychometric characteristics of the items were 
comparable between aboriginal and non-aboriginal examinees. The measures of internal 
consistency and item discrimination were quite similar for both groups in Mathematics, Science, and 
Social Studies. However, it is clear that the mean differences between the aboriginals and non- 
aboriginal samples are quite large. Non-aboriginals (M = 35.63, SD = 8.19) outperformed aboriginals 
(M = 22.43, SD = 9.12) on the Mathematics tests. The same trend was found on the Science test 
with non-aboriginals (M = 34.69, SD = 8.21) outperforming aboriginals (M = 22.45, SD = 8.10) and 
on the Social Studies test with non-aboriginals (M = 35.48, SD = 7.71) outperforming aborignals (M = 
22.30, SD = 8.26). The aboriginal distributions are more positively skewed and leptokurtic than the 
non-aboriginal distributions which appear quite normal in shape. Additionally, mean item difficulties 
illustrate that items, on average, were more difficult for aboriginal examinees compared to their non- 
aboriginal peers. 
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Confirmatory DIF Analyses 

Items on the Mathematics, Science, and Social Studies achievement tests, which were expected 
to favor aboriginals or non-aboriginals, were initially tested with a single-item confirmatory DIF 
analysis. On the Mathematics test, one of the items predicted to favor aboriginals reached statistical 
significant while none of the items predicted to favor non-aboriginals reached significance. Only one 
of the 14 items predicted to favor aboriginals and one of the four items predicted to favor non- 
aborginals on the Science test were statistically significant. Finally, for the Social Studies test, only 
one of the six items predicted to favor aboriginals reached statistical significance. 

Confirmatory DBF Analyses 

A summary of the DBF analyses for the Mathematics, Science, and Social Studies tests is found 
in Table 2. The statistical test for the items predicted to favor aboriginals on the Mathematics test 

yielded a non-significant /3 ml of -0.03 (g =0.70). Similarly, the test for the items predicted to favor 
non-aboriginals on the Mathematics tests yielded a non-significant of 0.02 (p = 0.67). Those 
items suspected to favor aboriginals on the Science test resulted in a non-significant j 3^ of 0.04 (p 

= 0.84) while those items suspected to favor non-aboriginals produced a significant (3 mI of -0.12 (p 
=0.05) in favor of aboriginals. Those items suspected of favor aboriginals on the Social Studies test 
yielded a non-significant j8 w/ of -0.09 (p = 0.20) while those items suspected to favor non- 
aboriginals produced a non-significant of -0.04 (g = 0.31). 

Conclusions and Discussion 

Researchers have found that achievement scores for aboriginal examinees are consistently lower 
than those of non-aboriginals. Consistent with this research, we found systematic group differences 
favoring non-aboriginals in each content area using overall achievement test scores. To date, 
however, researchers have failed to address the role cognitive processes may play in this outcome. 
The goal of the present study was to examine the performance of aboriginal and non-aboriginal 
students on Grade 6 Mathematics, Science, and Social Studies achievement tests. Four raters 
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trained in the PASS theory coded achievement test items in each content area as eliciting either 
simultaneous or successive processing skills. We found that most items identified by the raters as 
favoring either aboriginals or non-aboriginals due to simultaneous or successive processing skills 
failed to reach statistical significance in the confirmatory DIF analyses. When bundled, those items 
also failed to reach statistical significance in the confirmatory DBF analyses. 

One key factor may account for the difference between the current study and previous studies — 
previous researchers have used “purer” measures of simultaneous processing. For example, 
Krywaniukand Das (1976) and Janzen (2000), using the Kaufman Assessment Battery for Children 
(K-ABC) and the Cognitive Assessment System (CAS), respectively, found that aboriginal 
examinees had strong simultaneous processing skills. However, both the K-ABC and the CAS 
contain scales that specifically measure simultaneous processing. In the current study, achievement 
test items were used and the items may possess greater levels of both simultaneous and successive 
processing — in other words, the achievement test items may not be an adequate measure of 
simultaneous and successive processing skills. Moreover, in the current study, raters were provided 
with a list of item features that are characteristic of simultaneous and successive processing. It is 
possible that only certain subcategories of simultaneous or successive processing characteristics 
favor aboriginals and non-aboriginals. For example Das, Manos, and Kanungo (1975) found that 
aboriginal children outperformed non-aboriginal children in figure copying — a spatial-simultaneous 
processing task— but that they performed equally as well as non-aboriginal students on a 
simultaneous task involving non-verbal reasoning. Unfortunately, there is very little additional 
research on this topic. In future research, it may be useful to identify those items that clearly 
exhibiting the processing skills in these subcategories (based on the limited research outcomes 
available to-date) to determine if they can predict which items will favor aboriginal or non-aboriginal 
examinees. Additionally, DBF analyses could be carried out on these "purer" assessment measures 
of simultaneous and successive processing skills to confirm the belief that aboriginals perform better 
on these measures of simultaneous processing. 

Three other limitations of the current study must also be noted. First, a small sample of aboriginal 
examinees were used. Small sample sizes are typical in many DIF analyses with minority focal 
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groups (Mazor, Clauser & Hambleton, 1992). Rogers (1989) found that by increasing the sample 
size for the focal group, substantial increases in power can be achieved (as cited in Mazor, Clauser 
& Hambleton, 1992). Mazor, Clauser, and Hambleton (1992) concluded that sample sizes of 500 
can yield accurate DIF results but larger samples are preferred. Although sample sizes for the focal 
group exceeded 500 in the content areas of Mathematics and Social Studies, the Grade 6 Science 
sample only contained 480 aboriginal examinees. 

Second, and more importantly, the ability distribution differences between the aboriginal and non- 
aboriginal sample were large. Shealy and Stout (1993) found inflated Type I error in the presence of 
latent ability distributional differences for SIBTEST when reference and focal groups were compared. 
Other researchers have also found that when the focal and reference distributions are markedly 
different, SIBTEST tends to have inflated Type I error rate and lower power because matching 
precision will depend on the extent to which the focal and reference groups differ in ability (Chang, 
Mazzeo & Roussos,1996; Clauser, Nungester, Mazor & Ripkey, 1996). Recently, Gotzmann, 
Boughton and Gierl (2001) reported that the Type I error rates for SIBTEST generally remained 
below the nominal rate when target ability differences were small. However, withlarger target ability 
differences of 1.5 and 2.0 standard deviations between the reference and focal groups, the Type I 
error rates increased markedly. The power rates for SIBTEST were generally quite high with no 
ability differences and moderate with large ability differences. Gotzmann et al. (2001) concluded that 
SIBTEST can be used with group ability differences as large as 1.5 standard deviations but only with 
large samples (e.g., at least 2000 examinees per group). 

To overcome this limitation, researchers could match examinees in those regions of the test 
score distributions that overlap and that have adequate sample sizes. In other words, DIF analyses 
could be based on moderate to high ability aboriginal examinees (i.e., those examinees one 
standard deviation above the mean or higher) with moderate to low ability non-aboriginal examinees 
(i.e., those examinees one standard deviation below the mean or lower). This approach would 
increase the power and decrease the Type I error rate for SIBTEST by refining the matching variable 
as it would only include those segments of the test score distributions that coincide. In may, 
however, lead to more difficult substantive interpretations because only a portion of the aboriginal 




14 



DBF Analyses 14 



and non-aboriginal sample Is represented In the analysis. We are currently pursuing this line of 
research. 

Third, aboriginal peoples are a diverse group with many languages and cultures. Much of the 
research demonstrating that aboriginal favor a simultaneous processing style has been conducted 
with Cree aboriginals. The current study sample included aboriginals from across the Canadian 
province of Alberta thereby encompassing a number of different First Nations such as Cree, Dene, 
and Blackfoot. Therefore, future researchers adopting a DIF approach should consider using a more 
homogeneous group of aboriginal examinees when attempting to link cognitive style and test 
performance. Unfortunately, sample size restrictions become a factor when more homogeneous 
groups are selected. Nevertheless, this approach may yield more conclusive findings. 

Other factors could also help explain the poorer performance of aboriginals on achievement tests. 
The current study examined cognitive style as a possible reason for this trend. However, many other 
factors have been identified including socio-economic status (e.g., Rogers, Wentzel, & Ndalichako, 
1997; Wood & Clay, 1996), language differences (Brescia & Fortune, 1989), cultural bias (Estrin & 
Nelson-Barber, 1995; Neely & Shaughnessy, 1987; Wood & Clay, 1996), and differences in test- 
taking skills (Brescia & Fortune, 1987). These variable should also be considered as explanations 
are sought to account for the discrepancy in achievement test scores between aboriginal and non- 
aboriginal examinees. 
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